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Although powerful computers have allowed complex physical and manmade hardware systems to be 
modelled successfully, we have encountered persistent problems with the reliably of computer 
models for systems involving human learning, human action, and human organizations. This is not 
misfortune: unlike physical and manmade sytems, human systems do not operate under a fixed set of 
laws - the rules governing the actions allowable in the system can be changed without warning at 
any moment and can evolve over time. That the governing laws are inherently unpredictable raises 
serious questions about the reliability of models when applied to human situations. In these domains, 
computers are better used, not for prediction and planning, but for aiding humans. Examples are 
systems that help humans speculate about possible futures, offer advice about possible actions in a 
domain, systems that gather information from the networks, and systems that track and support 
work flows in organizations. 
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Why don't we have a complete plan for reforms? In order to play chess, one 
must know the rules ... how to move the various pieces on the board. But it is 
not possible to know the situation on the chessboard after the 15th or 25th 

move. 

— Vaclav Klaus, Finance Minister, Czechoslovakia 

Each day, new scientists, managers, executives, and government leaders express 
concern about the safety and reliability of complex computer systems. As such systems 
take charge of everything from phone calls to flights, we are all exposed to a growing 
danger of man-made disasters. 

Important among complex computer systems are computer models used for 
simulations and predictions of phenomena in areas ranging from physics to hardware 
engineering to socio-economic systems. Computer models have become an area of 
concern unto themselves. Their misuse could lead governments to adopt disastrous 
policies in dealing with such subjects as global warming and global economic stability. 
The proliferation of computer models supporting divergent points of view— for example, 
computer simulations supporting conflicting theories of global warming or nuclear 
winter— can easily mislead the lay public. Models whose results depend on assumptions 
about human behavior are the most likely to produce controversial results. 

In early November 1990 the Association for Computing Machinery (ACM) brought 
together leading scientists, business executives, and government officials to discuss 
public -policy questions surrounding computer models. I will summarize the main points 
about modeling made by the principal speakers at that meeting. 
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Modeling Expertise 

Knowledge-based systems (KBSs) are important examples of computer models. A 
KBS is supposed to reproduce the decisions of an expert in a domain. KBSs have come 
under fire because to many observers the majority of them have fallen short of the 
promise of competent performance ( 3 ). 

John Kunz attributes part of the problem to a design-and-testing process taken from 
software engineering, a process that begins with a formal specification of the system’s 
behavior and ends with an acceptance test ( 9 ). This process cannot take into account that 
the standards for expert performance can shift as a field changes. Kunz argues that, to 
obtain reliable KBSs, continual testing and improvement must be the standard approach. 
The tests must do more than compare KBS decisions with real situations; they must 
validate that at all times the recommended actions fulfill the purpose of the system, that 
the reasoning procedures are valid for the domain, and that the recommended actions are 
consistently endorsed and assessed as competent by human experts. Kunz recommends 
that the tests include simple realistic cases as well as cases that apply various stresses to 
the KBS. He recommends that some of the tests be retrospective (comparing KBS 
decisions with those of experts in the past) and that some be prospective (measuring the 
performance of the KBS against that of experts in real time) before the system is 
deployed in the field. 

KBS’s are founded on the assumption that an expert works from a complete theory 
of the domain. Once a theory is articulated as a set of rules and stored in a database, the 
superior power of the computer can draw inferences much faster than the expert. That 
this has not been accomplished cannot be blamed on a lack of computing power, 
memory, research effort, or cooperation of experts. An explanation gaining credence is 
that experts themselves do not work from complete theories, and much of their expertise 
cannot be articulated in language. The advocates of neural networks claim they have 
found a way to overcome the inability to articulate expertise. Neural networks mimic the 
biological structure of the brain and therefore the expert’s approach to gathering and 
organizing information; once the networks have been trained, their advocates say, they 
will be able to acquire the knowledge experts have that cannot be articulated as rules. 

For this reason, neural networks have been offered as a model that differs from the 
traditional approach of mathematical modeling. 

At the ACM meeting Jay Forrester argued that all human decisions are taken with 
respect to (possibly subconscious) mental models, and that computers should be used to 
augment human mental-modeling powers ( 6 ). He is interested in models that make 
predictions about the future behavior of large organizations and societies. He maintains 
that human beings are notoriously inept at understanding the dynamics of systems that 
contain feedback control loops. Feedback loops, which are familiar features of 
mechanical systems and biological organisms, also permeate organizations and social 
systems. The modeling approach Forrester calls system dynamics is aimed at giving us a 
tool to aid in understanding the operation of systems for which we have only a static 
description. He claims that many organizations can be successfully modeled because the 
members of the organization follow policies that are either explicit or are pan of their 
habitual behavior; hence they can be stated as precise static rules that can be embodied as 
interacting functions in the model. Forrester has a good deal of optimism that socio- 
economic systems can ultimately be modeled and that system dynamics is a powerful 
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general approach. 


Limits of Modeling 

Stuart Dreyfus, a long-time advocate of modeling and critic of expert systems, is 
concerned that we understand the limits of modeling so that our claims about models can 
be well grounded ( 4 ). He argues that in most socio-economic domains, neither 
conventional mathematical modeling (including rule-based artificial intelligence) nor 
neural-network modeling are as trustworthy as the judgments of impartial, experienced 
experts. He calls the actions of experts in a domain a form of "skillful coping," about 
which there are four extant theories: 1) Expert behavior is an unconscious application of 
a conventional model. 2) It is uninterpretable neuronal and biochemical activity. 3) It is 
a process of recalling memories that match the current situation. 4) It is uninterpretable 
brain activity evolved from a domain theory learned during an initial formal encounter 
with the domain through some teacher. 

Dreyfus says that Forrester bases system dynamics on the first theory, whereas 
Dreyfus himself finds the fourth theory much more credible and consistent with evidence 
about skillful coping. He concludes that computers that provide facts and suggest 
decisions can improve the judgment of experienced people. In the hands of 
inexperienced people, however, such computers may actually degrade coping skill. 
Education that equates expertise with models can inhibit the development of good 
judgment 

Steve Kline draws a sharp distinction between physical systems and systems that 
include human beings ( 8 ). He uses a simple complexity index to demonstrate the 
qualitative differences between these two kinds of systems. His measure counts the 
number of variables, parameters and feedback loops in the system being modeled. 
Physical systems modeled by differential equations (e.g., fluid flows) have low model 
complexity (on the order of 10 1 ) and may have high computational complexity. 
Hardware systems (e.g., airplanes and computer networks) have moderate model 
complexity (on the order of 10 6 ) and moderate to high computational compexity. But 
models for "human systems"-brains, personalities, organizations, economies and 
societies-all have extremely high model compexity (on the order of 10 13 and beyond). 

In Kline’s analysis physical systems and hardware systems have three 
characteristics that lead to low model complexity: they operate under invariant rules, 
their parts are context-independent, and they are not self-observing. In contrast, human 
systems have changing rules and are context-dependent and self-observing. The key 
distinguishing factor is that major jumps in complexity arise when the "rules of the 
game" (the governing laws) can change or evolve unpredictably. This has important 
implications for models of human systems. They must be created by ongoing 
development rather than prior analysis. They cannot be used reliably for prediction; 
instead they must supplement and augment, but not replace, human judgment. Kline 
ends up questioning the "science-based" approach to modeling these systems, an 
approach rooted in the Newtonian (mechanistic) tradition, which assumes that all of the 
universe is governed by fixed laws. 
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Eleanor Wynn continues the skepticism toward computer models of human 
activities by questioning whether the perspective of information processing itself is 
sufficient to understand human systems (15). Noting the widespread agreement that we 
do not know how to design complex software systems that are dependable, she observes 
that most of the discussion about software occurs within the paradigm of software 
engineering that begins with a formal specification and ends with an acceptance test. She 
argues that this paradigm completely misses how good designs are made because it is 
context-independent and cannot take into account the perspectives of users. She calls 
attention to the Scandinavian paradigm of user-centered design, which has already 
yielded consistently effective designs. A new paradigm for design will not be easy to 
establish, she says, because it is closely related to the ideology of "scientific 
management", which regards jobs as optimized formal descriptions of responsibilities, 
human beings as resources that do jobs, and competitors as belligerent organizations. 


Description, Computation, Prediction 

These authors share the conclusions that models involving human behavior are 
unavoidably complex, that such models may not work except in limited cases, and that 
even then they will be made to work by ongoing development rather than by prior 
analysis. They suggest that one’s trust in the reliability of such models depends on one’s 
assumptions about how biological organisms and societies learn and act. But they 
diverge on this claim: Models can produce greater understanding of complex human 
phenomena, lead us to wise decisions and guide us to effective actions. Forrester is 
optimistic about this claim. Kunz implicitly accepts it in the domain of knowledge-based 
systems. But Dreyfus, Wynn, and Kline express serious doubts. The divergence of 
views on this important question is at the heart of the question of computer modeling of 
human realities. 

In what follows I offer my own analysis of this claim, and I suggest ways that 
computers can assist us effectively in the domain of human actions. 

What is a model? We usually understand a model to be a symbolic representation 
of a set of objects, their relationships and their allowable motions (14). We use models 
in three principal ways: 

Description. We sometimes use a model to describe how a system works. The 
formality of the description sharpens understanding; the description can be shared 
with others to achieve a shared understanding. Examples are a blueprint, a scale 
model of a railroad, the equations of motion of a planet, the scientific method, and 
the software-design process. 

Computation. We sometimes use a model to guide, to reproduce or to calculate 
action in the domain. Examples are following directions from an inertial guidance 
system (guiding), a flight simulator (reproducing) or computing a measurement 
(calculating). 

Prediction. We sometimes use a model to predict the future state of a system with 
tolerable certainty. Examples are models that predict the lift of a wing in flight, the 
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position of a star, or the future state of the weather or the world economy. A model 
is useful for prediction only if the future state can be calculated much more rapidly 
than in real time and its users agree that the assumptions about parameter values and 
governing laws will hold at the future time. 

These three aspects are hierarchical in the sense that prediction relies on a model to 
compute a future state given future values of parameters, and computation relies on a 
precise description of the allowable motions of a system. 

Models are of universal interest because of our unavoidable concern to anticipate 
and prepare for future action, and because they make the world seem simpler and more 
understandable. Description, computation and prediction are three ways in which we 
accomplish this. The case of a map illustrates. We can use a map to achieve an 
understanding of the layout of a city and to discuss possible tours with others 
(description). We can use a map to navigate through the city to a destination 
(computation). Or we can use a map to estimate how long it will take to reach a 
destination (prediction). And the aphorism, “The map is not the territory,” reminds us 
not to confuse models with reality. 


Reliability and Complexity 

What is reliability in modeling? A model is reliable if we find that it recurrently 
agrees with phenomena in the domain modeled. A model with many parameters is 
unlikely to be judged reliable because it is infeasible to explore the parameter space 
completely during testing and because the model’s calculations may be sensitive to small 
changes in an unknown few of the parameters. A model is also unlikely to be judged 
reliable if we have not found a set of variables sufficient to describe the phenomenon of 
interest. 

The more sophisticated predictive models provide indicators of the certainty of the 
prediction. These measures take the form of confidence intervals associated with 
numerical values or probabilities associated with states. If not interpreted properly, these 
measures can give a false sense of security about the reliability of the model— everyone 
has had experiences in which we were certain of an outcome that never happened. Some 
modelers say that these measures allow comparisons: a model with smaller confidence 
intervals than another would be judged as the more powerful. It is important to ask 
whether the model does significantly better than random guesses. Even if it does, it need 
not be reliable because the uncertainty in its predictions may be too great. 

The assessment of reliability is important in science and technology, where we seek 
to exploit recurrent phenomena. There are, however, important situations in which 
reliability is not an appropriate standard for assessment of a model. Forrester’s system 
dynamics, for example, is a form of modeling aimed at supporting human beings in 
grounding their speculations about the fate of systems in which the actors follow 
unchanging rules. 

What is complexity? Complexity is an assessment we make about our capacity to 
accurately describe, compute or predict phenomena in a domain. This assessment is 
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related to the number of variables, parameters and loops that exist in a system: for the 
greater those numbers, the greater our uncertainty about how the system works and the 
lower our capacity to describe, simulate or predict it accurately. 

Note that chaotic behavior in the sense recently understood as "mathematical chaos" 
is not judged as complex by this standard (5). Such behavior can be described by simple 
equations, and its future trajectory can be calculated by iteration. These mathematical 
tools and powerful computers now allow us to calculate in excellent detail phenomena 
that we used to call complex— examples include cloud formation, leaf structure and 
turbulence. Present computers are not fast enough for prediction— for example, recent 
joint studies of turbulence by investiagtors at Stanford and the NASA Ames Research 
Center took six months of time on a Cray Y-MP supercomputer for each case. On the 
other hand, chaotic functions do not necessarily provide reliable models because the 
future states can sometimes be very sensitive to the initial condition, about which there is 
often great uncertainty. 

It is worth noting that we can make separate assessments of complexity about a 
model and about the domain modeled. This is because the model is itself a system that 
has variables, parameters and loops. It is possible to offer a simple model for a complex 
domain, although we would be surprised if the model were reliable in this case. It is 
common to see complex models for simple domains. Our ideal is a simple model that 
reliably and rapidly reproduces the selected phenomena of the domain. 


Meta-modeling 

In an effort to understand where the complexity of models originates and how 
approximations arise, some modelers have modeled the modeling process itself. 
Agrawal’s book is an example (1). If you will permit me some light mathematics, I can 
show you how the modeling process itself introduces complexities that are often 
overlooked. 

One can regard the construction of a model as a series of steps, each of which 
transforms a model into a simpler model by introducing a simplifying assumption. Let 
us focus on one of these steps. Suppose we have a model M with parameters P and one 
variable x . The model can be used to calculate a value of its variable by an algorithm 
x = M(P ). Suppose now we seek a faster algorithm by introducing a simplifying 
assumption A that maps the values of the original parameters and variable into the new 
parameters P ' of M P ' = A (P jc ). The new model can now be used to calculate a value 
for the variable: x = M'(P') = M'(P x )). Notice that the calculation is of the form: x = 
F(x). The predicted value of the variable is now the fixed point of a nonlinear function. 
If the value of x is initially unknown, an iteration must be employed to find a convergent 
value, and the total computation time is not simply one application of the simpler model. 
Some of the fixed points of the function may be stable and others unstable, meaning that 
the final value may depend on the initial condition. Moreover, the time to convergence 
becomes an issue, and there is a possibility of chaotic behavior (in the mathematical 
sense) in the iteration. This situation gets worse when several variables of the original 
model participate in the simplifying assumptions. 
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The conclusion is that "simplifying assumptions" can introduce rather than resolve 
computational complexities, a possibility that looms larger for systems with many 
variables and for models with many simplifications. This means that an assessment of 
reliability may be extremely difficult to make for models that contain many simplifying 
assumptions. And, as Wynn points out, it is easy for us to ignore these complexities by 
pursuading ourselves that the model is real or that the simplifying assumptions are of no 
consequence. 


Meta-assessments 

In addition to assessments of reliability and complexity, we often make a third kind 
of assessment— a meta-assessment— about whether a model’s complexity or degree of 
reliability is "good" or "bad." I bring this up because in many discussions about 
modeling complex systems, I hear a background of frustration that the systems to be 
modeled, and thus the models themselves, are complex. It is "bad" that things are 
complex and a challenge to our ingenuity to find a reliable and computable model 
anyway. If we have such a meta-assessment, it will be extremely difficult to conclude 
that some systems are not worth an attempt at modeling. For example, many people 
accept that a major responsibility of government is to "plan" the economy, and thus it is 
necessary to have reliable models that will allow prediction of future states of the 
economy resulting from various policies, so that we can determine now which policies to 
enact. We seek a scientific approach to governance. In this context the absence of a 
reliable model of the world economy is "bad" and is sufficient to motivate the 
expenditure of millions of dollars in pursuit of computer models of the world’s economy. 

We do not always judge that complexity is "bad." We live in an unimaginably 
complex world of five billion people, each engaged in a network of conversations with 
others. Declarations made in distant parts of this network can affect the possibilities 
open to us even though we are not part of the conversation leading to the decision. (The 
Iraqi take over of Kuwait is a good recent example.) Most of us simply accept that the 
world network of human conversations is highly complex and unpredictable, that the 
rules of the game may be altered without warning at any time, and that the rules will 
surely evolve. Our strategy in this case is not to find models whose predictions can guide 
our actions; it is rather to create organizations and use their power to effect action. 
Successful organizations do not rely on computer models; they develop strategies to 
position themselves in the world marketplace. Entrepreneurs such as Tom Peters thrive 
in this environment of complexity and uncertainty-they assess complexity as "good" 
( 12 ). 

Another category of meta-assessments are those people make of the future as they 
carry out their work in organizations and social systems. We call these assessments 
"moods." Not only do individuals have moods, so do organizations and social units. 
Organizations with good moods (high morale) generally do better than those with bad 
moods, and one of the jobs of managers and executives is to generate good moods in 
their organizations ( 12 ). A country can enter a depression if enough people get into a 
mood of pessimism in which they hoard their money. The phenomenon of moods is very 
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important to the success of organizations. Our inability to predict moods adds to the 
complexities we face in making predictions about organizations and social systems. 


Evolving Rules 

Systems with many human participants may be so subject to changes of rules that 
their future trajectories cannot even be described as computable functions, much less 
predicted. It is often difficult or impossible to determine the variables that affect the 
phenomenon, even when we have many cases available for study-understanding human 
personalities or the collapse of complex societies ( 13 ) are examples. 

What drives us to seek models in the face of evidence that reliable computable 
models may not exist? We have all been brought up in a scientific world view, 
conditioned by 300 years of successful physics modeling, dating from the time of 
Newton, which inclines us to believe that all the world’s a mechanism, a clock that God 
created and left ticking. We tend to interpret the historical record as a demonstration that 
the world became a better place with the introduction of the rigor of science. We tend to 
believe that everything, including the human brain, the human personality, and human 
social systems, can in principle be modeled by a set of equations. Given enough research 
we can find the equations, and given enough computer resources we can solve them 
( 2 , 11 ). 

Our scientific tradition has a darker side. It views the world, including people, as a 
collection of resources to be acquired, used, optimized and discarded when no longer 
needed. It views situations, including those that involve the human condition, as 
"problems" for which technological and procedural "solutions" arc to be found; unable to 
admit that some problems may be insoluble, this discourse labels such problems as 
"intractable" but ultimately solvable given sufficient knowledge and resources. We need 
to ask ourselves whether our drive to model human complexities might not be an 
overextension of science, and whether our drive to use scientific models to solve world 
problems might not reflect the hubris of science. We need to ask ourselves whether some 
of the models of complex phenomena we seek to construct would gain us anything if we 
could find them. 

At bottom, a model is nothing more than an interpretation of the world. The 
invention of interpretations is a fundamentally human activity that is intimately involved 
with our understanding of truth. As scientists, we like to say that scientific laws and 
mathematical theorems already exist awaiting discovery. But if we carefully examine the 
processes of science, we find paradigms other than discovery. Roald Hoffmann says that 
the creation of new substances not found in nature is the dominant activity of disciplines 
such as chemistry and molecular biology ( 7 ). Bruno Latour goes farther, observing that 
in practice a statement is accepted as true by a community if no one has been able to 
produce evidence or an argument that persuades others to dissent ( 10 ). Science is a 
process of constructing facts, and different scientific communities can construct different 
systems of interpretation of the same physical phenomena. Western and Eastern 
medicine, for example, are two scientifically valid systems of interpretation about disease 
and human disorders; each recommends different interventions for the same symptoms 
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and sees phenomena that are invisible to the other, and their interpretations are not easily 
reconciled. 


Productive Uses of Computers 

Several conclusions emerge from the discussion above: 

• As part of our modeling efforts we must come to understand the domains over 
which a given model is reliable, partly reliable and unreliable. We must also 
understand the situations in which models can be useful as a way of grounding 
speculations about the future dynamics of systems. 

• Systems whose rules can evolve or change in unpredictable ways are unlikely to 
have a reliable predictive or speculative model. 

• We must be careful with the output of models, being constantly skeptical that those 
outputs are "facts" or are accurate descriptions of the world. 

• In our technological age, it is easy to accept the claim that every phenomenon can 
ultimately be modeled, given sufficient knowledge and computational resources. 
There is reason to doubt this faith. 

• If our mood makes us disinclined to acceptance complexity, it is easy to substitute 
the model for reality and to confuse our opinions with "scientific facts" supported by 
the model. 

In spite of my questions and doubts, I accept that in limited domains we may be 
able to find reliable predictive models of systems in which human beings participate. At 
this point, however, we have no consensus on where the limits are. 

If we cannot model human systems, what can we do with computers? We can use 
them to augment human capacities, especially in those areas where we are limited, 
notably in processing power and in memory. We can use KBSs as advisors to suggest 
actions based on analysis of past situations, and let the current decision taken by the 
human being become another data point for future analysis. We can use the worldwide 
network of computers to gather information about what is going on in the realms of 
interest to our organizations. We can use computers to help manage and track the flow of 
work and information. We can confine models to domains in which their predictive 
power can be used reliably, namely domains in which the rules are known in advance. In 
all cases, however, we must let the computer support the decision-maker, and not let the 
computer make the decisions. 
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